Phase 2 begins: stop managing your data by anecdote and learn to pull the numbers yourself — starting with the pandas mental model
Day 31 of 66
Phase 1 made you fluent in the craft: rubrics, agreement, preference data, multimodal evaluation. You can produce gold-standard human data and measure it. Phase 2 is about running the operation — and the first thing an operation runs on is numbers. This week is the technical fluency that turns "I think our quality dropped" into "annotator C's mean score fell 0.6 points last week on safety items; here's the table."
A Human Data Lead who has to ask someone else to pull the numbers is leading blind and slow. The person who can load the label export, group it, and find the disagreement hot-spots in five minutes sets the agenda. You can't lead what you can't query. So this week you learn to query.
You don't need to become a data scientist. You need a small, sharp toolkit: pandas for ad-hoc analysis of label files on your laptop, SQL for the big datasets a lab stores in a warehouse, and the handful of statistics that actually matter for judging data. Today is pandas — the tool you'll live in.
Forget the jargon. pandas gives you one core object — the DataFrame — and a DataFrame is a table with named columns and rows, exactly like a spreadsheet or a database table. Your label export is already a table: one row per label, with columns like item_id, annotator, score, task. pandas just lets you operate on that table in code instead of by hand.
There are really only four moves you need this week, and you already think in all four — you just do them in a spreadsheet today:
pd.read_csv("labels.csv") reads a CSV off disk into a DataFrame. That's the whole first step. df.head() shows you the top few rows so you can see what you're working with — always your first command on a new file.
df[df.score < 3] keeps only rows where the score is below 3. This is "show me the failures" — the single most common thing a lead asks of a label set.
df.groupby("annotator") splits the table into one bucket per annotator. This is the move that turns 10,000 labels into a one-line-per-person profile. Tomorrow's lesson lives here.
.agg(["count", "mean", "std"]) says "for each group, give me the count, the average, and the spread." Group + aggregate together answer almost every question you'll have about who labeled what and how.
Here is the smallest possible end-to-end example. Read every line — this is the shape of nearly everything you'll write this week.
import pandas as pd
df = pd.read_csv("labels.csv") # the whole file becomes one table
print(df.head()) # look before you leap: top 5 rows
low = df[df.score < 3] # filter: only the failing labels
print(f"{len(low)} labels scored below 3")
print(df.groupby("annotator")["score"].mean()) # avg score per personread_csv loads, head previews, the bracket filter keeps failing rows, and groupby(...)["score"].mean() gives you each annotator's average in one line. Notice df.groupby("annotator")["score"] reads almost like English: "group by annotator, then look at the score column."Excel is fine for 200 rows you'll look at once. But a lead's analysis must be repeatable — same script, next week, new file, identical answer. A four-line script you can rerun on every weekly export beats a spreadsheet you rebuild by hand and quietly get wrong. Code is how you stop managing by anecdote.
The whole point isn't the syntax — it's the independence. When you can answer your own questions about the data, three things change. You stop waiting on an analyst for numbers you need today. You catch problems (a drifting annotator, a hot-spot item) before they compound. And when you bring a finding to your manager, it's a table with your name on it, not a vibe.
Install pandas (pip install pandas) and work the official "10 minutes to pandas" tutorial from the getting-started page. Then load any CSV you have — even a label export from your own work — and run df.head(), one filter, and one groupby(...).mean(). The goal is comfort with the four moves, not mastery.
import pandas as pd runs without error.read_csv and run df.head(). Note the column names out loud.An IC asks the analyst to pull the numbers and waits a day for the answer. A lead loads the export and answers their own question in five minutes — which is what lets them set the agenda instead of reacting to someone else's table.
Say this in an interview: "I don't manage my data by anecdote. When I say quality dropped, I can hand you the DataFrame — annotator, week, and the exact items — because I pulled it myself before the meeting."